• 1 Preface
  • 2 Questions for the Group Based on My Preliminary Analysis Below
  • 3 Required Packages
  • 4 Extracting COVID-19 Data
  • 5 Plotting the Data for Saint Louis
    • 5.1 Differencing and Focusing on the Data Starting from Jan 04, 2021
  • 6 Creating time splits for Training and Validation
  • 7 Training Different Time-Series Models
    • 7.1 Model Building
    • 7.2 Training Performance
      • 7.2.1 Plotting the Residuals
      • 7.2.2 Statistical Tests for the Residuals
      • 7.2.3 Model’s Training Performance
  • 8 Testing Performance
    • 8.1 Holdout Performance
      • 8.1.1 Plotting the Residuals
      • 8.1.2 Statistical Tests for the Residuals
      • 8.1.3 Model’s Holdout Performance

1 Preface

Some R stuff:

  • You can download the entire Rmd for this Markdown by clicking on the upper right Code button.
  • You can also use the upper right Code button to hide all code if you just want to focus on the analysis.

Given that we cannot access the data and I was under the impression that our coding walk-through today was somewhat helpful, I attempted to create a template for the analysis that you can use for the data. I was not able to access hospitalizations on the county level from the Covid19 DataHub so I will be using deaths inlieu of hospitalizations for this walk-through.

This template makes the following assumptions:

  • The COVID-19 DataHub provides the data at three units of aggregation: country, state, and county. From my understanding, you are modeling data for Greater St. Louis (i.e., St. Louis MSA). To get the data similar to what Peter has access to, I summed both cases and deaths for counties in both Illinois (‘Bond’, ‘Calhoun’, ‘Clinton’, ‘Jersey’, ‘Macoupin’, ‘Madison’, ‘Monroe’) and Missouri (‘Crawford’, ‘Franklin’, ‘Jefferson’, ‘Lincoln’, ‘St. Charles’, ‘St. Clair’, ‘St. Louis’, ‘St. Louis City’, ‘Warren’) based on the Greater St. Louis Wikipedia Page.

  • On top of my head, I did not remember the study period that you had for the analysis. So I have used data starting from Jan 04, 2021 (first Monday of the Year) for this analysis.

  • For the St. Louis Health Department, I assumed that the non_weekend holidays matched those of the New York Stock Exchange. These were extracted using HOLIDAY_SEQUENCE(start_date = '2021-01-04', end_date = '2022-08-01', calendar = 'NYSE') from the tidyquant package. The exact holiday dates from this function call are: 2021-01-18, 2021-02-15, 2021-04-02, 2021-05-31, 2021-07-05, 2021-09-06, 2021-11-25, 2021-12-24, 2022-01-17, 2022-02-21, 2022-04-15, 2022-05-30, 2022-07-04.

  • I kind of eyeballed the dates for the dominant COVID-19 variant based on the following NY Times Article. I have used the plot within the article to define the dominant variant to be as follows: dominant_variant = case_when(date < ymd('2021-03-01') ~ 'Epsilon', date < ymd('2021-06-15') ~ 'Alpha', date < ymd('2021-12-15') ~ 'Delta', date >= ymd('2021-12-15') ~ 'Omicron') %>% as_factor(). This allowed me to take into account the ‘entire’ time-series rather than just the Omicron part.

  • I assumed that one of the end goals from your analysis would be forecasting future hospitalizations. So far, in our meetings, we discussed explanatory modeling. So I divided my data into a training and a holdout sample – primarily for the purpose of showing you how its done. Obviously, you can override this by ignoring the holdout and just focusing on the training results (which you can also set to be the entire time period of interest).

  • I did not lag the confirmed cases. I believe you should test out different lags within the context of both the auto.arima() and the other models since we would be hypothesizing that the lagged confirmed cases would lead to future hospitalizations. It is my understanding that the auto.arima() does not account for the lagged relationships on the exogenous variables. See the Lagged Predictors in the fpp2 book. The implications of introducing the lagged predictors on the current code are as follows:

    • The lag can be computed as follows within the mutate function: confirmed1 = lag(confirmed, k = 1) for lag1. Changing k would change the lag.
    • The lag should also be accounted for in the initial_time_split() to ensure a seperation between the training and testing dataset. In that function, the default value for lag = 0, and you would need to change it to the highest lag you are evaluating.

  • I ignored potentially relevant predictors such as the stringency index and vaccines. In my opinion, these can be important with the different waves. This would be nice to explore as you continue to build the model.

2 Questions for the Group Based on My Preliminary Analysis Below

  • How should we handle negative counts in cases/deaths/hospitalizations? See the death count on March 29, 2021 in the plot in Section 5.1 as an example.

  • Should we worry about the residuals’ departure from normality? See the results of the Shapiro-Wilk Test in Section 7.2.2.

  • Do you agree with my comment about lagged predictors? See the second to last assumption in the Preface section.

3 Required Packages

The code chunk below contains the packages used in this analysis

# tidyverse pkg for data manipulation
if(!require(tidyverse)) {install.packages('tidyverse'); library(tidyverse)}

# readr for reading files
if(!require(readr)) {install.packages('readr'); library(readr)}

# timetk for tidy time-series preprocessing
if(!require(timetk)) {install.packages('timetk'); library(timetk)}

# tidyquant for tidy time-series analysis
if(!require(tidyquant)) {install.packages('tidyquant'); library(tidyquant)}

# modeltime for tidy forecasting
if(!require(modeltime)) {install.packages('modeltime'); library(modeltime)}

# for interactive plots
if(!require(plotly)) {install.packages('plotly'); library(plotly)}

# for tidy machine learning
if(!require(tidymodels)) {install.packages('tidymodels'); library(tidymodels)}

# for the xgboost model
if(!require(xgboost)) {install.packages('xgboost'); library(xgboost)}

4 Extracting COVID-19 Data

Per the Preface, I read the data used in this analysis from the COVID-19 DataHub. To not require the installing of the COVID19 package, I have downloaded the zip file and unzipped using R. Then, I filtered the data to Missouri and Illinois, selected the appropriate counties, and computed the variables that we are interested in.

# creating a temp file for downloading the data
temp = tempfile() 

# download the file to temporary location
download.file("https://storage.covid19datahub.io/country/USA.csv.zip", temp)

# unzip and read the file
covid_tbl = unz(temp, "USA.csv") %>% 
  # reading the data from the CSV
  read_csv() %>% 
  # filtering to Missouri and Illinois
  filter(administrative_area_level_2 %in% c('Illinois', 'Missouri') )


st_louis_tbl = covid_tbl %>% 
  # Relevant Counties in Illnois
  filter( 
    (administrative_area_level_2 == 'Illinois' &
       administrative_area_level_3 %in% c('Bond', 'Calhoun', 'Clinton', 'Jersey', 'Macoupin', 'Madison', 'Monroe') ) | 
      # Relevant Counties in Missouri 
      (administrative_area_level_2 == 'Missouri' &
         administrative_area_level_3 %in% c('Crawford', 'Franklin', 'Jefferson', 'Lincoln', 'St. Charles', 'St. Clair', 'St. Louis', 'St. Louis City', 'Warren'))
  )

# Aggregating the counts by day (so that we have an approximation of total numbers for st. louis)
st_louis_agg_tbl = 
  st_louis_tbl %>% 
  # grouping by date so we can created an aggregated summation across all counties
  group_by(date) %>% 
  select(date, 
         # variables to be summed across all counties in St. Louis
         confirmed, deaths, recovered, hosp, icu, vent, 
         # population will be summed only to see if the aggregation is reasonable
         # i.e., is the total pop close to the actual for the St. Louis MSA
         population,
         # other variables that can be included in the analysis later
         stringency_index) %>% 
  summarise_at(
    vars(confirmed, deaths, recovered, hosp, icu, vent, population),
    .funs = sum, na.rm = T
    ) %>% 
  ungroup() 

# remove temp file
unlink(temp) 

# glimpse of the data
glimpse(st_louis_agg_tbl)
## Rows: 878
## Columns: 8
## $ date       <date> 2020-03-07, 2020-03-08, 2020-03-09, 2020-03-10, 2020-03-11…
## $ confirmed  <dbl> 1, 1, 1, 1, 1, 1, 2, 2, 3, 5, 7, 12, 20, 26, 40, 77, 94, 99…
## $ deaths     <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 3, 3,…
## $ recovered  <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
## $ hosp       <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
## $ icu        <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
## $ vent       <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
## $ population <dbl> 994205, 994205, 994205, 994205, 994205, 994205, 994205, 994…

From the printout of the data, we have 8 variables and 878 days of data. Note that the dominant_variant was eyeballed as mentioned previously and the holidays variable assumes that your health department matches the holidays taken by the NYSE.

5 Plotting the Data for Saint Louis

From the plot below, it is clear that this public dataset does not contain information on hospitalizations, vent, and ICU. Thus, I will model deaths as a function of cases. Peter, you can change the variables based on your dataset.

It is also clear that the data is cumulative so we will difference to have the daily new cases and deaths.

st_louis_agg_tbl %>% 
  pivot_longer(cols = c(confirmed, deaths, recovered, hosp, icu, vent, population),
               names_to = 'statistic') %>% 
  ggplot(
    aes(x = date, y = value)
  ) + 
  geom_line() +
  facet_wrap(~ statistic, scales = 'free_y', ncol = 1) + 
  theme_bw() +
  scale_x_date(breaks = scales::pretty_breaks(n = 12)) +
  scale_y_continuous(labels = scales::comma) +
  labs(title = 'Plots of the Time-Series of Potential Variables for St. Louis MSA',
       subtitle = 'Hosp, ICU, Recovered and Vent have no data',
       caption = 'Based on data aggregated from the COVID19 DataHub',
       x = 'Date',
       y = NULL)-> p1

ggplotly(p1)
0200,000400,000600,00002,0004,0006,0008,000-0.050-0.0250.0000.0250.050-0.050-0.0250.0000.0250.0501,000,0001,500,0002,000,0002,500,000-0.050-0.0250.0000.0250.050Apr 2020Jul 2020Oct 2020Jan 2021Apr 2021Jul 2021Oct 2021Jan 2022Apr 2022Jul 2022-0.050-0.0250.0000.0250.050
Plots of the Time-Series of Potential Variables for St. Louis MSADateconfirmeddeathshospicupopulationrecoveredvent

5.1 Differencing and Focusing on the Data Starting from Jan 04, 2021

Based on our explanatory analysis of the data, there seemed to be some large outliers in the beginning of 2021. So I am starting the analysis from the first Monday of the year. I differenced to compute the variables of interest. The code below plots the two time-series of interest (confirmed cases and confirmed deaths), color coded based on the COVID-19 dominant variant.

st_louis_agg_tbl = 
  st_louis_agg_tbl %>% 
  # keeping only relevant columns for forecasting
  select(date, confirmed, deaths) %>% 
  # converting the cumulative counts to new
  mutate(
    confirmed = confirmed - lag(confirmed),
    deaths = deaths - lag(deaths) 
  ) %>% 
  filter(date >= ymd('2021-01-04')) %>% 
  mutate(
    # creating a data frame of variants based on 
    # https://www.nytimes.com/interactive/2021/health/coronavirus-variant-tracker.html
    dominant_variant = 
      case_when(
        date < ymd('2021-03-01') ~ 'Epsilon',
        date < ymd('2021-06-15') ~ 'Alpha',
        date < ymd('2021-12-15') ~ 'Delta',
        date >= ymd('2021-12-15') ~ 'Omicron'
      ) %>% as_factor(),
    # creating a list of special holidays
         holidays = 
      if_else(date %in% HOLIDAY_SEQUENCE(start_date = min(date),
                                     end_date = max(date),
                                     calendar = 'NYSE'),
              true = 'yes',
              false = 'no') %>% as_factor()
           ) %>% 
  drop_na()

st_louis_agg_tbl %>% 
  pivot_longer(cols = c(confirmed, deaths),
               names_to = 'statistic') %>% 
  ggplot(
    aes(x = date, y = value, color = dominant_variant)
  ) + 
  geom_line() +
  facet_wrap(~ statistic, scales = 'free_y', ncol = 1) + 
  theme_bw() +
  scale_x_date(breaks = scales::pretty_breaks(n = 12)) + 
  scale_y_continuous(labels = scales::comma) +
  scale_color_brewer(palette = 'Paired') +
  labs(x = 'Date',
       y = 'New Daily Counts', 
       title = 'Plots of Confirmed Cases and Deaths for St. Louis MSA') -> p2

ggplotly(p2)
05,00010,000Jan 2021Apr 2021Jul 2021Oct 2021Jan 2022Apr 2022Jul 2022050100150
dominant_variantEpsilonAlphaDeltaOmicronPlots of Confirmed Cases and Deaths for St. Louis MSADateNew Daily Countsconfirmeddeaths

An important question to address (if this is in Peter’s data) is how do we handle negative counts. A naive fix in R would be to use the max of the value and 0 (see the pmax()). However, this ignores the misreporting of the data.

6 Creating time splits for Training and Validation

Below, I have capitalized on the initial_time_split() to split the time-series such that the first 90% of the data are used for model building and the remaining 10% are used for validation. We can redo this if we solely want to focus on the explanatory modeling component.

splits = initial_time_split(st_louis_agg_tbl, prop = 0.9)

# Printing out the splits so that we know the number of observations used for model training
splits
## <Training/Testing/Total>
## <517/58/575>
# Training data start and end dates
paste('The starting and ending dates for training are',
      splits$data[splits$in_id, 'date'] %>% head(n=1) %>% pull(),
      'and', 
      splits$data[splits$in_id, 'date'] %>% tail(n=1) %>% pull(),
      'respectively. For the holdout data, the starting and trainig dates are',
      splits$data[splits$out_id, 'date'] %>% head(n=1) %>% pull(),
      'and', 
      splits$data[splits$out_id, 'date'] %>% tail(n=1) %>% pull())
## [1] "The starting and ending dates for training are 2021-01-04 and 2022-06-04 respectively. For the holdout data, the starting and trainig dates are 2022-06-05 and 2022-08-01"

7 Training Different Time-Series Models

In this section, I have quickly trained the following five models:

  • A univariate Auto ARIMA model with no xreg
  • An Auto ARIMA model with confirmed, holidays (NYSE holidays) and dominant variant as our xreg
  • An Auto Arima Model with xgboost (using the xregs) on the Arima errors
  • The Prophet Model, originally developed by Facebook. See the Forecasting at Scale Paper for more details.
  • A linear regression model with predictors as.numeric(date) + confirmed + holidays + dominant_variant

I have capitalized on the modeltime package vignette to quickly run these models. Note that the modeltime API also allows for running other machine learning models for time-series applications.

7.1 Model Building

# a basic univariate ARIMA model using “Auto Arima” using arima_reg()
# using the modeltime pkg this will automatically pick the weekly seasonality
model_fit_arima =arima_reg() %>%
    set_engine(engine = "auto_arima") %>%
    fit(deaths ~ date, data = training(splits) )
## frequency = 7 observations per 1 week
# ARIMA with xreg
model_fit_arima_xreg = arima_reg() %>%
  set_engine(engine = "auto_arima") %>%
  # confirmed, holidays and dominant variant as our xreg
  fit(
    deaths ~ date + confirmed + holidays + dominant_variant,
      data = training(splits) 
    )
## frequency = 7 observations per 1 week
# Auto Arima Model with xgboost on the Arima errors
model_fit_arima_boosted = arima_boost(
    min_n = 2,
    learn_rate = 0.015
) %>%
    set_engine(engine = "auto_arima_xgboost") %>%
    fit(deaths ~ date + confirmed + holidays + dominant_variant,
        data = training(splits) )
## frequency = 7 observations per 1 week
# the prophet model used by Facebook
model_fit_prophet = prophet_reg() %>%
  set_engine(engine = "prophet") %>%
  fit(deaths ~ date + confirmed + holidays + dominant_variant,
      data = training(splits) )
## Disabling yearly seasonality. Run prophet with yearly.seasonality=TRUE to override this.
## Disabling daily seasonality. Run prophet with daily.seasonality=TRUE to override this.
# Linear Regression 
model_fit_lm = linear_reg() %>%
    set_engine("lm") %>%
    fit(deaths ~ as.numeric(date) + confirmed + holidays + dominant_variant,
        data = training(splits) )

models_tbl = modeltime_table(
    model_fit_arima,
    model_fit_arima_xreg,
    model_fit_arima_boosted,
    model_fit_prophet,
    model_fit_lm
)

models_tbl
## # Modeltime Table
## # A tibble: 5 × 3
##   .model_id .model   .model_desc                                  
##       <int> <list>   <chr>                                        
## 1         1 <fit[+]> ARIMA(2,1,3)(2,0,0)[7]                       
## 2         2 <fit[+]> REGRESSION WITH ARIMA(1,0,0)(1,0,0)[7] ERRORS
## 3         3 <fit[+]> ARIMA(5,1,1)(0,0,2)[7] W/ XGBOOST ERRORS     
## 4         4 <fit[+]> PROPHET W/ REGRESSORS                        
## 5         5 <fit[+]> LM

If this data is close to your data, you can see that the first non-seasonal difference was taken in both auto.arima() applications. This answers the question from Allison today.

7.2 Training Performance

7.2.1 Plotting the Residuals

The code below is used to plot the residuals for each of the models. The first three lines of code are very similar to the code shown in the package vignette. The additional code is to allow you to customize the plot to your liking.

models_tbl %>%
    modeltime_calibrate(new_data = training(splits)) %>%
    modeltime_residuals() %>%
    plot_modeltime_residuals(.interactive = FALSE,
                             .type = 'timeplot') +
  scale_x_date(breaks = scales::pretty_breaks(12)) +
  scale_color_brewer(palette = 'Dark2') +
  facet_wrap(~ .model_desc, ncol = 1, scales = 'free') +
  theme_bw() +
  theme(legend.position = 'none') +
  labs(
    title = 'Residuals plot for the five models based on our training data',
    subtitle = 'The residuals are large on the same day irrespective of model')

7.2.2 Statistical Tests for the Residuals

The modeltime_residuals_test() computes the results from 4 different statistical tests:

  • Shapiro-Wilk Test tests the Normality of the residuals. The Null Hypothesis is that the residuals are normally distributed. A low p-value below a given significance level indicates the values are NOT Normally Distributed.

  • Both the Box-Pierce and Ljung-Box Tests are used to test for the absence of autocorrelation in residuals. A low p-value below a given significance level indicates the values are autocorrelated.

models_tbl %>%
    modeltime_calibrate(new_data = training(splits)) %>%
    modeltime_residuals() %>%
  modeltime_residuals_test()
## # A tibble: 5 × 6
##   .model_id .model_desc          shapiro_wilk box_pierce ljung_box durbin_watson
##       <int> <chr>                       <dbl>      <dbl>     <dbl>         <dbl>
## 1         1 ARIMA(2,1,3)(2,0,0)…     1.59e-29    0.880     0.880            2.01
## 2         2 REGRESSION WITH ARI…     4.54e-31    0.957     0.957            1.99
## 3         3 ARIMA(5,1,1)(0,0,2)…     2.34e-28    0.786     0.786            2.02
## 4         4 PROPHET W/ REGRESSO…     4.34e-30    0.0586    0.0579           1.83
## 5         5 LM                       1.07e-30    0.00212   0.00205          1.73
# If you want to explore the residuals seperately, they are stored as follows
training_residuals = models_tbl %>%
    modeltime_calibrate(new_data = training(splits)) %>%
    modeltime_residuals()

The results above indicate that the residuals are not normally distributed. However, they also indicate that no significant autocorrelation is exhibited in the residuals as indicated by both the Box-Pierce and Ljung-Box test results.

7.2.3 Model’s Training Performance

Multiple accuracy measures can be computed as follows.

models_tbl %>%
    modeltime_calibrate(new_data = training(splits)) %>%
    modeltime_accuracy() %>%
    table_modeltime_accuracy(
    )
.model_id
.model_desc
.type
mae
mape
mase
smape
rmse
rsq
1
ARIMA(2,1,3)(2,0,0)[7]
Fitted
7.56
Inf
0.77
95.11
13.51
0.23
2
REGRESSION WITH ARIMA(1,0,0)(1,0,0)[7] ERRORS
Fitted
7.56
Inf
0.77
96.51
13.57
0.22
3
ARIMA(5,1,1)(0,0,2)[7] W/ XGBOOST ERRORS
Fitted
7.5
Inf
0.76
95.73
12.84
0.3
4
PROPHET W/ REGRESSORS
Fitted
6.98
Inf
0.71
101.11
12.65
0.32
5
LM
Test
8.4
Inf
0.85
102.62
14.21
0.14

Note that MAPE is infinite in the below table since there are several days where the count of new deaths = 0. The MASE indicates that all models improve over the naive forecast by about 25%. However, the rsq for all models is low. May be we need to add additional predictors.

8 Testing Performance

The name of the function from the modeltime package is somewhat confusing. It is called calibrate but calibrating adds a new column, .calibration_data, with the test predictions and residuals inside. A few notes on Calibration:

  • Calibration is how confidence intervals and accuracy metrics are determined
  • Calibration Data is simply forecasting predictions and residuals that are calculated from out-of-sample data.
  • After calibrating, the calibration data follows the data through the forecasting workflow.
calibration_tbl = models_tbl %>%
    modeltime_calibrate( new_data = testing(splits) )

calibration_tbl
## # Modeltime Table
## # A tibble: 5 × 5
##   .model_id .model   .model_desc                          .type .calibration_da…
##       <int> <list>   <chr>                                <chr> <list>          
## 1         1 <fit[+]> ARIMA(2,1,3)(2,0,0)[7]               Test  <tibble>        
## 2         2 <fit[+]> REGRESSION WITH ARIMA(1,0,0)(1,0,0)… Test  <tibble>        
## 3         3 <fit[+]> ARIMA(5,1,1)(0,0,2)[7] W/ XGBOOST E… Test  <tibble>        
## 4         4 <fit[+]> PROPHET W/ REGRESSORS                Test  <tibble>        
## 5         5 <fit[+]> LM                                   Test  <tibble>

Per the description above the code chunk, the model descriptions did not change and we only appended the testing data.

8.1 Holdout Performance

8.1.1 Plotting the Residuals

models_tbl %>%
    modeltime_calibrate(new_data = testing(splits)) %>%
    modeltime_residuals() %>%
    plot_modeltime_residuals(.interactive = FALSE,
                             .type = 'timeplot') +
  scale_x_date(breaks = scales::pretty_breaks(12)) +
  scale_color_brewer(palette = 'Dark2') +
  facet_wrap(~ .model_desc, ncol = 1) +
  theme_bw() +
  theme(legend.position = 'none') +
  labs(
    title = 'Residuals plot for the five models based on our holdout data',
    subtitle = 'The ARIMA type models are most accurate. LM is over-predicting since residuals are mostly negative')

8.1.2 Statistical Tests for the Residuals

models_tbl %>%
    modeltime_calibrate(new_data = testing(splits)) %>%
    modeltime_residuals() %>%
  modeltime_residuals_test()
## # A tibble: 5 × 6
##   .model_id .model_desc          shapiro_wilk box_pierce ljung_box durbin_watson
##       <int> <chr>                       <dbl>      <dbl>     <dbl>         <dbl>
## 1         1 ARIMA(2,1,3)(2,0,0)…  0.00000149      0.215     0.203          1.97 
## 2         2 REGRESSION WITH ARI…  0.000156        0.0494    0.0438         0.432
## 3         3 ARIMA(5,1,1)(0,0,2)…  0.000000585     0.475     0.464          2.10 
## 4         4 PROPHET W/ REGRESSO…  0.00680         0.148     0.138          1.59 
## 5         5 LM                    0.0000257       0.215     0.204          0.382

The holdout results are consistent with the observations made on the training data. One exception is that the ARIMA with xreg (model 2) shows correlated residuals on the holdout dataset.

8.1.3 Model’s Holdout Performance

Note that MAPE is infinite in the below table since there are several days where the count of new deaths = 0

models_tbl %>%
    modeltime_calibrate(new_data = testing(splits)) %>%
    modeltime_accuracy() %>%
    table_modeltime_accuracy(
    )
.model_id
.model_desc
.type
mae
mape
mase
smape
rmse
rsq
1
ARIMA(2,1,3)(2,0,0)[7]
Test
3.16
Inf
0.92
133.58
3.67
0.09
2
REGRESSION WITH ARIMA(1,0,0)(1,0,0)[7] ERRORS
Test
6.53
Inf
1.9
149.04
7
0.35
3
ARIMA(5,1,1)(0,0,2)[7] W/ XGBOOST ERRORS
Test
2.75
Inf
0.8
135.4
3.51
0.08
4
PROPHET W/ REGRESSORS
Test
6.4
Inf
1.86
160.86
7.33
0.02
5
LM
Test
6.59
Inf
1.92
149.29
7.01
0.37

The first and third models continue to be the best models. However, their improvement over the naive forecast (i.e., random walk) is smaller than the training dataset.

  1. Email: | Phone: +1-513-529-4185 | Website: Miami University Official↩︎

LS0tDQp0aXRsZTogIk1vZGVsaW5nIENPVklELTE5IERlYXRocyBpbiBTYWludCBMb3VpcyBhcyBhIEZ1bmN0aW9uIG9mIENhc2VzIg0KYXV0aG9yOg0KICAtIG5hbWU6ICJGYWRlbCBNLiBNZWdhaGVkIF5bRW1haWw6IGZtZWdhaGVkQG1pYW1pb2guZWR1IHwgUGhvbmU6ICsxLTUxMy01MjktNDE4NSB8IFdlYnNpdGU6IDxhIGhyZWY9XCJodHRwczovL21pYW1pb2guZWR1L2ZzYi9kaXJlY3RvcnkvP3VwPS9kaXJlY3RvcnkvbWVnYWhlZm1cIj5NaWFtaSBVbml2ZXJzaXR5IE9mZmljaWFsPC9hPl0iDQogICAgYWZmaWxpYXRpb246IEZhcm1lciBTY2hvb2wgb2YgQnVzaW5lc3MsIE1pYW1pIFVuaXZlcnNpdHkNCmRhdGU6ICJgciBmb3JtYXQoU3lzLnRpbWUoKSwgJyVCICVkLCAlWScpYCINCm91dHB1dDogDQogIGh0bWxfZG9jdW1lbnQ6DQogICAgY3NzOiBodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20vZm1lZ2FoZWQvdmFjY2luZXMvbWFpbi9Db2RlL2N1c3RvbS5jc3MNCiAgICBjb2RlX2ZvbGRpbmc6IHNob3cNCiAgICBjb2RlX2Rvd25sb2FkOiBUUlVFDQogICAgbnVtYmVyX3NlY3Rpb25zOiBUUlVFDQogICAgcGFnZWRfZGY6IFRSVUUNCiAgICB0b2M6IFRSVUUNCiAgICB0b2NfZmxvYXQ6IFRSVUUNCiAgICB0aGVtZTogcmVhZGFibGUNCiAgaW5jbHVkZXM6DQogICAgaW5faGVhZGVyOiBodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20vZm1lZ2FoZWQvdmFjY2luZXMvbWFpbi9Db2RlL3N0cnVjdHVyZS50ZXgNCi0tLQ0KDQoNCg0KYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9DQprbml0cjo6b3B0c19jaHVuayRzZXQoY2FjaGUgPSBUUlVFLA0KICAgICAgICAgICAgICAgICAgICAgIGVjaG8gPSBUUlVFLA0KICAgICAgICAgICAgICAgICAgICAgIHdhcm5pbmcgPSBGQUxTRSwNCiAgICAgICAgICAgICAgICAgICAgICBtZXNzYWdlID0gRkFMU0UsDQogICAgICAgICAgICAgICAgICAgICAgcHJvZ3Jlc3MgPSBGQUxTRSwgDQogICAgICAgICAgICAgICAgICAgICAgdmVyYm9zZSA9IEZBTFNFLA0KICAgICAgICAgICAgICAgICAgICAgIGRldiA9ICdwbmcnLA0KICAgICAgICAgICAgICAgICAgICAgIGZpZy5yZXRpbmEgPSAyLA0KICAgICAgICAgICAgICAgICAgICAgIG91dC53aWR0aCA9ICcxMDAlJykNCg0KIyBpbnN0YWxsaW5nIHRoZSByZW1vdGVzIGFuZCBpY29ucyBwYWNrYWdlcyBpZiBuZWVkZWQNCmlmKCFyZXF1aXJlKHJlbW90ZXMpKSB7aW5zdGFsbC5wYWNrYWdlcygncmVtb3RlcycpOyBsaWJyYXJ5KHJlbW90ZXMpfQ0KDQppZighcmVxdWlyZShpY29ucykpew0KICByZW1vdGVzOjppbnN0YWxsX2dpdGh1YigibWl0Y2hlbGxvaGFyYXdpbGQvaWNvbnMiKQ0KICBsaWJyYXJ5KGljb25zKQ0KICBkb3dubG9hZF9mb250YXdlc29tZSgpDQogIGRvd25sb2FkX2FjYWRlbWljb25zKCkNCn0gDQoNCg0KIyBTZXR0aW5nIHRoZSByYW5kb20gc2VlZCBhbmQgY2h1bmsgZGVwZW5kZW5jaWVzDQprbml0cjo6b3B0c19jaHVuayRzZXQoY2FjaGUuZXh0cmEgPSBzZXQuc2VlZCgyMDIyKSwNCiAgICAgICAgICAgICAgICAgICAgICBhdXRvZGVwID0gVFJVRSkgDQoNCg0KYGBgDQoNCiMgUHJlZmFjZSANCg0KKipTb21lIFIgc3R1ZmY6KiogICANCg0KLSBZb3UgY2FuIGRvd25sb2FkIHRoZSBlbnRpcmUgUm1kIGZvciB0aGlzIE1hcmtkb3duIGJ5IGNsaWNraW5nIG9uIHRoZSB1cHBlciByaWdodCBgQ29kZWAgYnV0dG9uLiAgDQotIFlvdSBjYW4gYWxzbyB1c2UgdGhlIHVwcGVyIHJpZ2h0IGBDb2RlYCBidXR0b24gdG8gaGlkZSBhbGwgY29kZSBpZiB5b3UganVzdCB3YW50IHRvIGZvY3VzIG9uIHRoZSBhbmFseXNpcy4gIA0KDQpHaXZlbiB0aGF0IHdlIGNhbm5vdCBhY2Nlc3MgdGhlIGRhdGEgYW5kIEkgd2FzIHVuZGVyIHRoZSBpbXByZXNzaW9uIHRoYXQgb3VyIGNvZGluZyB3YWxrLXRocm91Z2ggdG9kYXkgd2FzIHNvbWV3aGF0IGhlbHBmdWwsIEkgYXR0ZW1wdGVkIHRvIGNyZWF0ZSBhIHRlbXBsYXRlIGZvciB0aGUgYW5hbHlzaXMgdGhhdCB5b3UgY2FuIHVzZSBmb3IgdGhlIGRhdGEuIEkgd2FzIG5vdCBhYmxlIHRvIGFjY2VzcyBgaG9zcGl0YWxpemF0aW9uc2Agb24gdGhlIGNvdW50eSBsZXZlbCBmcm9tIHRoZSBbQ292aWQxOSBEYXRhSHViXShodHRwczovL2NvdmlkMTlkYXRhaHViLmlvLykgc28gSSB3aWxsIGJlIHVzaW5nIGBkZWF0aHNgIGlubGlldSBvZiBgaG9zcGl0YWxpemF0aW9uc2AgZm9yIHRoaXMgd2Fsay10aHJvdWdoLiANCg0KVGhpcyAqKnRlbXBsYXRlIG1ha2VzIHRoZSBmb2xsb3dpbmcgYXNzdW1wdGlvbnM6KiogIA0KDQotIFRoZSBDT1ZJRC0xOSBEYXRhSHViIHByb3ZpZGVzIHRoZSBkYXRhIGF0IHRocmVlIHVuaXRzIG9mIGFnZ3JlZ2F0aW9uOiBgY291bnRyeWAsIGBzdGF0ZWAsIGFuZCBgY291bnR5YC4gRnJvbSBteSB1bmRlcnN0YW5kaW5nLCB5b3UgYXJlIG1vZGVsaW5nIGRhdGEgZm9yIEdyZWF0ZXIgU3QuIExvdWlzIChpLmUuLCBTdC4gTG91aXMgTVNBKS4gVG8gZ2V0IHRoZSBkYXRhIHNpbWlsYXIgdG8gd2hhdCBQZXRlciBoYXMgYWNjZXNzIHRvLCBJIHN1bW1lZCBib3RoIGNhc2VzIGFuZCBkZWF0aHMgZm9yIGNvdW50aWVzIGluIGJvdGggSWxsaW5vaXMgKCdCb25kJywgJ0NhbGhvdW4nLCAnQ2xpbnRvbicsICdKZXJzZXknLCAnTWFjb3VwaW4nLCAnTWFkaXNvbicsICdNb25yb2UnKSBhbmQgTWlzc291cmkgKCdDcmF3Zm9yZCcsICdGcmFua2xpbicsICdKZWZmZXJzb24nLCAnTGluY29sbicsICdTdC4gQ2hhcmxlcycsICdTdC4gQ2xhaXInLCAnU3QuIExvdWlzJywgJ1N0LiBMb3VpcyBDaXR5JywgJ1dhcnJlbicpIGJhc2VkIG9uIHRoZSBbR3JlYXRlciBTdC4gTG91aXMgV2lraXBlZGlhIFBhZ2VdKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0dyZWF0ZXJfU3QuX0xvdWlzKS4gICANCg0KLSBPbiB0b3Agb2YgbXkgaGVhZCwgSSBkaWQgbm90IHJlbWVtYmVyIHRoZSBzdHVkeSBwZXJpb2QgdGhhdCB5b3UgaGFkIGZvciB0aGUgYW5hbHlzaXMuIFNvIEkgaGF2ZSB1c2VkIGRhdGEgc3RhcnRpbmcgZnJvbSBgSmFuIDA0LCAyMDIxYCAoZmlyc3QgTW9uZGF5IG9mIHRoZSBZZWFyKSBmb3IgdGhpcyBhbmFseXNpcy4gIA0KDQotIEZvciB0aGUgU3QuIExvdWlzIEhlYWx0aCBEZXBhcnRtZW50LCBJIGFzc3VtZWQgdGhhdCB0aGUgbm9uX3dlZWtlbmQgaG9saWRheXMgbWF0Y2hlZCB0aG9zZSBvZiB0aGUgTmV3IFlvcmsgU3RvY2sgRXhjaGFuZ2UuIFRoZXNlIHdlcmUgZXh0cmFjdGVkIHVzaW5nICBgSE9MSURBWV9TRVFVRU5DRShzdGFydF9kYXRlID0gJzIwMjEtMDEtMDQnLCBlbmRfZGF0ZSA9ICcyMDIyLTA4LTAxJywgY2FsZW5kYXIgPSAnTllTRScpYCBmcm9tIHRoZSBbdGlkeXF1YW50IHBhY2thZ2VdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3dlYi9wYWNrYWdlcy90aWR5cXVhbnQvdGlkeXF1YW50LnBkZikuIFRoZSBleGFjdCBob2xpZGF5IGRhdGVzIGZyb20gdGhpcyBmdW5jdGlvbiBjYWxsIGFyZTogYHIgdGlkeXF1YW50OjpIT0xJREFZX1NFUVVFTkNFKHN0YXJ0X2RhdGUgPSAnMjAyMS0wMS0wNCcsIGVuZF9kYXRlID0gCScyMDIyLTA4LTAxJywgY2FsZW5kYXIgPSAnTllTRScpYC4gIA0KDQotIEkga2luZCBvZiBgZXllYmFsbGVkIHRoZSBkYXRlcyBmb3IgdGhlIGRvbWluYW50IENPVklELTE5IHZhcmlhbnRgIGJhc2VkIG9uIHRoZSBbZm9sbG93aW5nIE5ZIFRpbWVzIEFydGljbGVdKGh0dHBzOi8vd3d3Lm55dGltZXMuY29tL2ludGVyYWN0aXZlLzIwMjEvaGVhbHRoL2Nvcm9uYXZpcnVzLXZhcmlhbnQtdHJhY2tlci5odG1sKS4gSSBoYXZlIHVzZWQgdGhlIHBsb3Qgd2l0aGluIHRoZSBhcnRpY2xlIHRvIGRlZmluZSB0aGUgZG9taW5hbnQgdmFyaWFudCB0byBiZSBhcyBmb2xsb3dzOiBgZG9taW5hbnRfdmFyaWFudCA9IGNhc2Vfd2hlbihkYXRlIDwgeW1kKCcyMDIxLTAzLTAxJykgfiAnRXBzaWxvbicsIGRhdGUgPCB5bWQoJzIwMjEtMDYtMTUnKSB+ICdBbHBoYScsIGRhdGUgPCB5bWQoJzIwMjEtMTItMTUnKSB+ICdEZWx0YScsIGRhdGUgPj0geW1kKCcyMDIxLTEyLTE1JykgfiAnT21pY3JvbicpICU+JSBhc19mYWN0b3IoKWAuIFRoaXMgYWxsb3dlZCBtZSB0byB0YWtlIGludG8gYWNjb3VudCB0aGUgJ2VudGlyZScgdGltZS1zZXJpZXMgcmF0aGVyIHRoYW4ganVzdCB0aGUgYE9taWNyb25gIHBhcnQuICANCg0KLSBJIGFzc3VtZWQgdGhhdCBvbmUgb2YgdGhlIGVuZCBnb2FscyBmcm9tIHlvdXIgYW5hbHlzaXMgd291bGQgYmUgKipmb3JlY2FzdGluZyoqIGZ1dHVyZSBob3NwaXRhbGl6YXRpb25zLiBTbyBmYXIsIGluIG91ciBtZWV0aW5ncywgd2UgZGlzY3Vzc2VkIGV4cGxhbmF0b3J5IG1vZGVsaW5nLiBTbyBJIGRpdmlkZWQgbXkgZGF0YSBpbnRvIGEgdHJhaW5pbmcgYW5kIGEgaG9sZG91dCBzYW1wbGUgLS0gcHJpbWFyaWx5IGZvciB0aGUgcHVycG9zZSBvZiBzaG93aW5nIHlvdSBob3cgaXRzIGRvbmUuIDxzcGFuIHN0eWxlPSJjb2xvcjpyZWQ7Zm9udC13ZWlnaHQ6Ym9sZCI+T2J2aW91c2x5LCB5b3UgY2FuIG92ZXJyaWRlIHRoaXMgYnkgaWdub3JpbmcgdGhlIGhvbGRvdXQgYW5kIGp1c3QgZm9jdXNpbmcgb24gdGhlIHRyYWluaW5nIHJlc3VsdHMgKHdoaWNoIHlvdSBjYW4gYWxzbyBzZXQgdG8gYmUgdGhlIGVudGlyZSB0aW1lIHBlcmlvZCBvZiBpbnRlcmVzdCkuPC9zcGFuPiAgDQoNCi0gKipJIGRpZCBub3QgbGFnIHRoZSBjb25maXJtZWQgY2FzZXMuKiogSSBiZWxpZXZlIHlvdSBzaG91bGQgdGVzdCBvdXQgZGlmZmVyZW50IGxhZ3Mgd2l0aGluIHRoZSBjb250ZXh0IG9mIGJvdGggdGhlICoqYXV0by5hcmltYSgpKiogYW5kIHRoZSBvdGhlciBtb2RlbHMgc2luY2Ugd2Ugd291bGQgYmUgaHlwb3RoZXNpemluZyB0aGF0IHRoZSBsYWdnZWQgY29uZmlybWVkIGNhc2VzIHdvdWxkIGxlYWQgdG8gZnV0dXJlIGhvc3BpdGFsaXphdGlvbnMuICoqSXQgaXMgbXkgdW5kZXJzdGFuZGluZyB0aGF0IHRoZSBhdXRvLmFyaW1hKCkgZG9lcyBub3QgYWNjb3VudCBmb3IgdGhlIGxhZ2dlZCByZWxhdGlvbnNoaXBzIG9uIHRoZSBleG9nZW5vdXMgdmFyaWFibGVzLioqIFNlZSB0aGUgW0xhZ2dlZCBQcmVkaWN0b3JzIGluIHRoZSBmcHAyIGJvb2tdKGh0dHBzOi8vb3RleHRzLmNvbS9mcHAyL2xhZ2dlZC1wcmVkaWN0b3JzLmh0bWwpLiA8c3BhbiBzdHlsZT0iY29sb3I6cmVkO2ZvbnQtd2VpZ2h0OmJvbGQiPlRoZSBpbXBsaWNhdGlvbnMgb2YgaW50cm9kdWNpbmcgdGhlIGxhZ2dlZCBwcmVkaWN0b3JzIG9uIHRoZSBjdXJyZW50IGNvZGUgYXJlIGFzIGZvbGxvd3M6PC9zcGFuPiAgIA0KICAqIFRoZSBsYWcgY2FuIGJlIGNvbXB1dGVkIGFzIGZvbGxvd3Mgd2l0aGluIHRoZSBtdXRhdGUgZnVuY3Rpb246IA0KICBgY29uZmlybWVkMSA9IGxhZyhjb25maXJtZWQsIGsgPSAxKWAgZm9yIGxhZzEuIENoYW5naW5nIGBrYCB3b3VsZCBjaGFuZ2UgdGhlIGxhZy4gIA0KICAqIFRoZSBsYWcgc2hvdWxkIGFsc28gYmUgYWNjb3VudGVkIGZvciBpbiB0aGUgYGluaXRpYWxfdGltZV9zcGxpdCgpYCB0byBlbnN1cmUgYSBzZXBlcmF0aW9uIGJldHdlZW4gdGhlIHRyYWluaW5nIGFuZCB0ZXN0aW5nIGRhdGFzZXQuIEluIHRoYXQgZnVuY3Rpb24sIHRoZSBkZWZhdWx0IHZhbHVlIGZvciBgbGFnID0gMGAsIGFuZCB5b3Ugd291bGQgbmVlZCB0byBjaGFuZ2UgaXQgdG8gdGhlIGhpZ2hlc3QgbGFnIHlvdSBhcmUgZXZhbHVhdGluZy4gDQoNCi0gSSBpZ25vcmVkIHBvdGVudGlhbGx5IHJlbGV2YW50IHByZWRpY3RvcnMgc3VjaCBhcyB0aGUgYHN0cmluZ2VuY3kgaW5kZXhgIGFuZCBgdmFjY2luZXNgLiBJbiBteSBvcGluaW9uLCB0aGVzZSBjYW4gYmUgaW1wb3J0YW50IHdpdGggdGhlIGRpZmZlcmVudCB3YXZlcy4gPHNwYW4gc3R5bGU9ImNvbG9yOnJlZDtmb250LXdlaWdodDpib2xkIj5UaGlzIHdvdWxkIGJlIG5pY2UgdG8gZXhwbG9yZSBhcyB5b3UgY29udGludWUgdG8gYnVpbGQgdGhlIG1vZGVsLjwvc3Bhbj4gDQoNCg0KLS0tDQoNCiMgUXVlc3Rpb25zIGZvciB0aGUgR3JvdXAgQmFzZWQgb24gTXkgUHJlbGltaW5hcnkgQW5hbHlzaXMgQmVsb3cNCg0KLSAqKkhvdyBzaG91bGQgd2UgaGFuZGxlIG5lZ2F0aXZlIGNvdW50cyBpbiBjYXNlcy9kZWF0aHMvaG9zcGl0YWxpemF0aW9ucz8qKiBTZWUgdGhlIGRlYXRoIGNvdW50IG9uIGBNYXJjaCAyOSwgMjAyMWAgaW4gdGhlIHBsb3QgaW4gU2VjdGlvbiA1LjEgYXMgYW4gZXhhbXBsZS4gIA0KDQotICoqU2hvdWxkIHdlIHdvcnJ5IGFib3V0IHRoZSByZXNpZHVhbHMnIGRlcGFydHVyZSBmcm9tIG5vcm1hbGl0eT8qKiBTZWUgdGhlIHJlc3VsdHMgb2YgdGhlIGBTaGFwaXJvLVdpbGsgVGVzdGAgaW4gU2VjdGlvbiA3LjIuMi4gDQoNCi0gKipEbyB5b3UgYWdyZWUgd2l0aCBteSBjb21tZW50IGFib3V0IGxhZ2dlZCBwcmVkaWN0b3JzPyoqIFNlZSB0aGUgc2Vjb25kIHRvIGxhc3QgYXNzdW1wdGlvbiBpbiB0aGUgUHJlZmFjZSBzZWN0aW9uLg0KDQoNCi0tLQ0KDQoNCiMgUmVxdWlyZWQgUGFja2FnZXMNCg0KVGhlIGNvZGUgY2h1bmsgYmVsb3cgY29udGFpbnMgdGhlIHBhY2thZ2VzIHVzZWQgaW4gdGhpcyBhbmFseXNpcw0KDQpgYGB7ciBwa2dzfQ0KIyB0aWR5dmVyc2UgcGtnIGZvciBkYXRhIG1hbmlwdWxhdGlvbg0KaWYoIXJlcXVpcmUodGlkeXZlcnNlKSkge2luc3RhbGwucGFja2FnZXMoJ3RpZHl2ZXJzZScpOyBsaWJyYXJ5KHRpZHl2ZXJzZSl9DQoNCiMgcmVhZHIgZm9yIHJlYWRpbmcgZmlsZXMNCmlmKCFyZXF1aXJlKHJlYWRyKSkge2luc3RhbGwucGFja2FnZXMoJ3JlYWRyJyk7IGxpYnJhcnkocmVhZHIpfQ0KDQojIHRpbWV0ayBmb3IgdGlkeSB0aW1lLXNlcmllcyBwcmVwcm9jZXNzaW5nDQppZighcmVxdWlyZSh0aW1ldGspKSB7aW5zdGFsbC5wYWNrYWdlcygndGltZXRrJyk7IGxpYnJhcnkodGltZXRrKX0NCg0KIyB0aWR5cXVhbnQgZm9yIHRpZHkgdGltZS1zZXJpZXMgYW5hbHlzaXMNCmlmKCFyZXF1aXJlKHRpZHlxdWFudCkpIHtpbnN0YWxsLnBhY2thZ2VzKCd0aWR5cXVhbnQnKTsgbGlicmFyeSh0aWR5cXVhbnQpfQ0KDQojIG1vZGVsdGltZSBmb3IgdGlkeSBmb3JlY2FzdGluZw0KaWYoIXJlcXVpcmUobW9kZWx0aW1lKSkge2luc3RhbGwucGFja2FnZXMoJ21vZGVsdGltZScpOyBsaWJyYXJ5KG1vZGVsdGltZSl9DQoNCiMgZm9yIGludGVyYWN0aXZlIHBsb3RzDQppZighcmVxdWlyZShwbG90bHkpKSB7aW5zdGFsbC5wYWNrYWdlcygncGxvdGx5Jyk7IGxpYnJhcnkocGxvdGx5KX0NCg0KIyBmb3IgdGlkeSBtYWNoaW5lIGxlYXJuaW5nDQppZighcmVxdWlyZSh0aWR5bW9kZWxzKSkge2luc3RhbGwucGFja2FnZXMoJ3RpZHltb2RlbHMnKTsgbGlicmFyeSh0aWR5bW9kZWxzKX0NCg0KIyBmb3IgdGhlIHhnYm9vc3QgbW9kZWwNCmlmKCFyZXF1aXJlKHhnYm9vc3QpKSB7aW5zdGFsbC5wYWNrYWdlcygneGdib29zdCcpOyBsaWJyYXJ5KHhnYm9vc3QpfQ0KYGBgDQoNCg0KLS0tDQoNCiMgRXh0cmFjdGluZyBDT1ZJRC0xOSBEYXRhDQoNClBlciB0aGUgUHJlZmFjZSwgSSByZWFkIHRoZSBkYXRhIHVzZWQgaW4gdGhpcyBhbmFseXNpcyBmcm9tIHRoZSBbQ09WSUQtMTkgRGF0YUh1Yl0oaHR0cHM6Ly9jb3ZpZDE5ZGF0YWh1Yi5pby9hcnRpY2xlcy9kYXRhLmh0bWwpLiBUbyBub3QgcmVxdWlyZSB0aGUgaW5zdGFsbGluZyBvZiB0aGUgYENPVklEMTlgIHBhY2thZ2UsIEkgaGF2ZSBkb3dubG9hZGVkIHRoZSB6aXAgZmlsZSBhbmQgdW56aXBwZWQgdXNpbmcgUi4gVGhlbiwgSSBmaWx0ZXJlZCB0aGUgZGF0YSB0byBNaXNzb3VyaSBhbmQgSWxsaW5vaXMsIHNlbGVjdGVkIHRoZSBhcHByb3ByaWF0ZSBjb3VudGllcywgYW5kIGNvbXB1dGVkIHRoZSB2YXJpYWJsZXMgdGhhdCB3ZSBhcmUgaW50ZXJlc3RlZCBpbi4NCg0KYGBge3IgcmF3X2RhdGF9DQojIGNyZWF0aW5nIGEgdGVtcCBmaWxlIGZvciBkb3dubG9hZGluZyB0aGUgZGF0YQ0KdGVtcCA9IHRlbXBmaWxlKCkgDQoNCiMgZG93bmxvYWQgdGhlIGZpbGUgdG8gdGVtcG9yYXJ5IGxvY2F0aW9uDQpkb3dubG9hZC5maWxlKCJodHRwczovL3N0b3JhZ2UuY292aWQxOWRhdGFodWIuaW8vY291bnRyeS9VU0EuY3N2LnppcCIsIHRlbXApDQoNCiMgdW56aXAgYW5kIHJlYWQgdGhlIGZpbGUNCmNvdmlkX3RibCA9IHVueih0ZW1wLCAiVVNBLmNzdiIpICU+JSANCiAgIyByZWFkaW5nIHRoZSBkYXRhIGZyb20gdGhlIENTVg0KICByZWFkX2NzdigpICU+JSANCiAgIyBmaWx0ZXJpbmcgdG8gTWlzc291cmkgYW5kIElsbGlub2lzDQogIGZpbHRlcihhZG1pbmlzdHJhdGl2ZV9hcmVhX2xldmVsXzIgJWluJSBjKCdJbGxpbm9pcycsICdNaXNzb3VyaScpICkNCg0KDQpzdF9sb3Vpc190YmwgPSBjb3ZpZF90YmwgJT4lIA0KICAjIFJlbGV2YW50IENvdW50aWVzIGluIElsbG5vaXMNCiAgZmlsdGVyKCANCiAgICAoYWRtaW5pc3RyYXRpdmVfYXJlYV9sZXZlbF8yID09ICdJbGxpbm9pcycgJg0KICAgICAgIGFkbWluaXN0cmF0aXZlX2FyZWFfbGV2ZWxfMyAlaW4lIGMoJ0JvbmQnLCAnQ2FsaG91bicsICdDbGludG9uJywgJ0plcnNleScsICdNYWNvdXBpbicsICdNYWRpc29uJywgJ01vbnJvZScpICkgfCANCiAgICAgICMgUmVsZXZhbnQgQ291bnRpZXMgaW4gTWlzc291cmkgDQogICAgICAoYWRtaW5pc3RyYXRpdmVfYXJlYV9sZXZlbF8yID09ICdNaXNzb3VyaScgJg0KICAgICAgICAgYWRtaW5pc3RyYXRpdmVfYXJlYV9sZXZlbF8zICVpbiUgYygnQ3Jhd2ZvcmQnLCAnRnJhbmtsaW4nLCAnSmVmZmVyc29uJywgJ0xpbmNvbG4nLCAnU3QuIENoYXJsZXMnLCAnU3QuIENsYWlyJywgJ1N0LiBMb3VpcycsICdTdC4gTG91aXMgQ2l0eScsICdXYXJyZW4nKSkNCiAgKQ0KDQojIEFnZ3JlZ2F0aW5nIHRoZSBjb3VudHMgYnkgZGF5IChzbyB0aGF0IHdlIGhhdmUgYW4gYXBwcm94aW1hdGlvbiBvZiB0b3RhbCBudW1iZXJzIGZvciBzdC4gbG91aXMpDQpzdF9sb3Vpc19hZ2dfdGJsID0gDQogIHN0X2xvdWlzX3RibCAlPiUgDQogICMgZ3JvdXBpbmcgYnkgZGF0ZSBzbyB3ZSBjYW4gY3JlYXRlZCBhbiBhZ2dyZWdhdGVkIHN1bW1hdGlvbiBhY3Jvc3MgYWxsIGNvdW50aWVzDQogIGdyb3VwX2J5KGRhdGUpICU+JSANCiAgc2VsZWN0KGRhdGUsIA0KICAgICAgICAgIyB2YXJpYWJsZXMgdG8gYmUgc3VtbWVkIGFjcm9zcyBhbGwgY291bnRpZXMgaW4gU3QuIExvdWlzDQogICAgICAgICBjb25maXJtZWQsIGRlYXRocywgcmVjb3ZlcmVkLCBob3NwLCBpY3UsIHZlbnQsIA0KICAgICAgICAgIyBwb3B1bGF0aW9uIHdpbGwgYmUgc3VtbWVkIG9ubHkgdG8gc2VlIGlmIHRoZSBhZ2dyZWdhdGlvbiBpcyByZWFzb25hYmxlDQogICAgICAgICAjIGkuZS4sIGlzIHRoZSB0b3RhbCBwb3AgY2xvc2UgdG8gdGhlIGFjdHVhbCBmb3IgdGhlIFN0LiBMb3VpcyBNU0ENCiAgICAgICAgIHBvcHVsYXRpb24sDQogICAgICAgICAjIG90aGVyIHZhcmlhYmxlcyB0aGF0IGNhbiBiZSBpbmNsdWRlZCBpbiB0aGUgYW5hbHlzaXMgbGF0ZXINCiAgICAgICAgIHN0cmluZ2VuY3lfaW5kZXgpICU+JSANCiAgc3VtbWFyaXNlX2F0KA0KICAgIHZhcnMoY29uZmlybWVkLCBkZWF0aHMsIHJlY292ZXJlZCwgaG9zcCwgaWN1LCB2ZW50LCBwb3B1bGF0aW9uKSwNCiAgICAuZnVucyA9IHN1bSwgbmEucm0gPSBUDQogICAgKSAlPiUgDQogIHVuZ3JvdXAoKSANCg0KIyByZW1vdmUgdGVtcCBmaWxlDQp1bmxpbmsodGVtcCkgDQoNCiMgZ2xpbXBzZSBvZiB0aGUgZGF0YQ0KZ2xpbXBzZShzdF9sb3Vpc19hZ2dfdGJsKQ0KYGBgDQoNCkZyb20gdGhlIHByaW50b3V0IG9mIHRoZSBkYXRhLCB3ZSBoYXZlIGByIG5jb2woc3RfbG91aXNfYWdnX3RibClgIHZhcmlhYmxlcyBhbmQgIGByIG5yb3coc3RfbG91aXNfYWdnX3RibClgIGRheXMgb2YgZGF0YS4gTm90ZSB0aGF0IHRoZSBgZG9taW5hbnRfdmFyaWFudGAgd2FzIGV5ZWJhbGxlZCBhcyBtZW50aW9uZWQgcHJldmlvdXNseSBhbmQgdGhlIGBob2xpZGF5c2AgdmFyaWFibGUgYXNzdW1lcyB0aGF0IHlvdXIgaGVhbHRoIGRlcGFydG1lbnQgbWF0Y2hlcyB0aGUgaG9saWRheXMgdGFrZW4gYnkgdGhlIE5ZU0UuDQoNCi0tLQ0KDQojIFBsb3R0aW5nIHRoZSBEYXRhIGZvciBTYWludCBMb3Vpcw0KDQpGcm9tIHRoZSBwbG90IGJlbG93LCBpdCBpcyBjbGVhciB0aGF0IHRoaXMgcHVibGljIGRhdGFzZXQgZG9lcyBub3QgY29udGFpbiBpbmZvcm1hdGlvbiBvbiBob3NwaXRhbGl6YXRpb25zLCB2ZW50LCBhbmQgSUNVLiBUaHVzLCBJIHdpbGwgbW9kZWwgZGVhdGhzIGFzIGEgZnVuY3Rpb24gb2YgY2FzZXMuICoqUGV0ZXIsIHlvdSBjYW4gY2hhbmdlIHRoZSB2YXJpYWJsZXMgYmFzZWQgb24geW91ciBkYXRhc2V0KiouDQoNCkl0IGlzIGFsc28gY2xlYXIgdGhhdCB0aGUgZGF0YSBpcyBjdW11bGF0aXZlIHNvIHdlIHdpbGwgZGlmZmVyZW5jZSB0byBoYXZlIHRoZSBkYWlseSBuZXcgY2FzZXMgYW5kIGRlYXRocy4NCg0KYGBge3IgZWRhfQ0Kc3RfbG91aXNfYWdnX3RibCAlPiUgDQogIHBpdm90X2xvbmdlcihjb2xzID0gYyhjb25maXJtZWQsIGRlYXRocywgcmVjb3ZlcmVkLCBob3NwLCBpY3UsIHZlbnQsIHBvcHVsYXRpb24pLA0KICAgICAgICAgICAgICAgbmFtZXNfdG8gPSAnc3RhdGlzdGljJykgJT4lIA0KICBnZ3Bsb3QoDQogICAgYWVzKHggPSBkYXRlLCB5ID0gdmFsdWUpDQogICkgKyANCiAgZ2VvbV9saW5lKCkgKw0KICBmYWNldF93cmFwKH4gc3RhdGlzdGljLCBzY2FsZXMgPSAnZnJlZV95JywgbmNvbCA9IDEpICsgDQogIHRoZW1lX2J3KCkgKw0KICBzY2FsZV94X2RhdGUoYnJlYWtzID0gc2NhbGVzOjpwcmV0dHlfYnJlYWtzKG4gPSAxMikpICsNCiAgc2NhbGVfeV9jb250aW51b3VzKGxhYmVscyA9IHNjYWxlczo6Y29tbWEpICsNCiAgbGFicyh0aXRsZSA9ICdQbG90cyBvZiB0aGUgVGltZS1TZXJpZXMgb2YgUG90ZW50aWFsIFZhcmlhYmxlcyBmb3IgU3QuIExvdWlzIE1TQScsDQogICAgICAgc3VidGl0bGUgPSAnSG9zcCwgSUNVLCBSZWNvdmVyZWQgYW5kIFZlbnQgaGF2ZSBubyBkYXRhJywNCiAgICAgICBjYXB0aW9uID0gJ0Jhc2VkIG9uIGRhdGEgYWdncmVnYXRlZCBmcm9tIHRoZSBDT1ZJRDE5IERhdGFIdWInLA0KICAgICAgIHggPSAnRGF0ZScsDQogICAgICAgeSA9IE5VTEwpLT4gcDENCg0KZ2dwbG90bHkocDEpDQpgYGANCg0KDQojIyBEaWZmZXJlbmNpbmcgYW5kIEZvY3VzaW5nIG9uIHRoZSBEYXRhIFN0YXJ0aW5nIGZyb20gSmFuIDA0LCAyMDIxDQoNCkJhc2VkIG9uIG91ciBleHBsYW5hdG9yeSBhbmFseXNpcyBvZiB0aGUgZGF0YSwgdGhlcmUgc2VlbWVkIHRvIGJlIHNvbWUgbGFyZ2Ugb3V0bGllcnMgaW4gdGhlIGJlZ2lubmluZyBvZiAyMDIxLiBTbyBJIGFtIHN0YXJ0aW5nIHRoZSBhbmFseXNpcyBmcm9tIHRoZSBmaXJzdCBNb25kYXkgb2YgdGhlIHllYXIuIEkgZGlmZmVyZW5jZWQgdG8gY29tcHV0ZSB0aGUgdmFyaWFibGVzIG9mIGludGVyZXN0LiBUaGUgY29kZSBiZWxvdyBwbG90cyB0aGUgdHdvIHRpbWUtc2VyaWVzIG9mIGludGVyZXN0IChgY29uZmlybWVkIGNhc2VzYCBhbmQgYGNvbmZpcm1lZCBkZWF0aHNgKSwgY29sb3IgY29kZWQgYmFzZWQgb24gdGhlIENPVklELTE5IGRvbWluYW50IHZhcmlhbnQuIA0KDQpgYGB7ciBlZGFfZGlmZn0NCnN0X2xvdWlzX2FnZ190YmwgPSANCiAgc3RfbG91aXNfYWdnX3RibCAlPiUgDQogICMga2VlcGluZyBvbmx5IHJlbGV2YW50IGNvbHVtbnMgZm9yIGZvcmVjYXN0aW5nDQogIHNlbGVjdChkYXRlLCBjb25maXJtZWQsIGRlYXRocykgJT4lIA0KICAjIGNvbnZlcnRpbmcgdGhlIGN1bXVsYXRpdmUgY291bnRzIHRvIG5ldw0KICBtdXRhdGUoDQogICAgY29uZmlybWVkID0gY29uZmlybWVkIC0gbGFnKGNvbmZpcm1lZCksDQogICAgZGVhdGhzID0gZGVhdGhzIC0gbGFnKGRlYXRocykgDQogICkgJT4lIA0KICBmaWx0ZXIoZGF0ZSA+PSB5bWQoJzIwMjEtMDEtMDQnKSkgJT4lIA0KICBtdXRhdGUoDQogICAgIyBjcmVhdGluZyBhIGRhdGEgZnJhbWUgb2YgdmFyaWFudHMgYmFzZWQgb24gDQogICAgIyBodHRwczovL3d3dy5ueXRpbWVzLmNvbS9pbnRlcmFjdGl2ZS8yMDIxL2hlYWx0aC9jb3JvbmF2aXJ1cy12YXJpYW50LXRyYWNrZXIuaHRtbA0KICAgIGRvbWluYW50X3ZhcmlhbnQgPSANCiAgICAgIGNhc2Vfd2hlbigNCiAgICAgICAgZGF0ZSA8IHltZCgnMjAyMS0wMy0wMScpIH4gJ0Vwc2lsb24nLA0KICAgICAgICBkYXRlIDwgeW1kKCcyMDIxLTA2LTE1JykgfiAnQWxwaGEnLA0KICAgICAgICBkYXRlIDwgeW1kKCcyMDIxLTEyLTE1JykgfiAnRGVsdGEnLA0KICAgICAgICBkYXRlID49IHltZCgnMjAyMS0xMi0xNScpIH4gJ09taWNyb24nDQogICAgICApICU+JSBhc19mYWN0b3IoKSwNCiAgICAjIGNyZWF0aW5nIGEgbGlzdCBvZiBzcGVjaWFsIGhvbGlkYXlzDQogICAgICAgICBob2xpZGF5cyA9IA0KICAgICAgaWZfZWxzZShkYXRlICVpbiUgSE9MSURBWV9TRVFVRU5DRShzdGFydF9kYXRlID0gbWluKGRhdGUpLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGVuZF9kYXRlID0gbWF4KGRhdGUpLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNhbGVuZGFyID0gJ05ZU0UnKSwNCiAgICAgICAgICAgICAgdHJ1ZSA9ICd5ZXMnLA0KICAgICAgICAgICAgICBmYWxzZSA9ICdubycpICU+JSBhc19mYWN0b3IoKQ0KICAgICAgICAgICApICU+JSANCiAgZHJvcF9uYSgpDQoNCnN0X2xvdWlzX2FnZ190YmwgJT4lIA0KICBwaXZvdF9sb25nZXIoY29scyA9IGMoY29uZmlybWVkLCBkZWF0aHMpLA0KICAgICAgICAgICAgICAgbmFtZXNfdG8gPSAnc3RhdGlzdGljJykgJT4lIA0KICBnZ3Bsb3QoDQogICAgYWVzKHggPSBkYXRlLCB5ID0gdmFsdWUsIGNvbG9yID0gZG9taW5hbnRfdmFyaWFudCkNCiAgKSArIA0KICBnZW9tX2xpbmUoKSArDQogIGZhY2V0X3dyYXAofiBzdGF0aXN0aWMsIHNjYWxlcyA9ICdmcmVlX3knLCBuY29sID0gMSkgKyANCiAgdGhlbWVfYncoKSArDQogIHNjYWxlX3hfZGF0ZShicmVha3MgPSBzY2FsZXM6OnByZXR0eV9icmVha3MobiA9IDEyKSkgKyANCiAgc2NhbGVfeV9jb250aW51b3VzKGxhYmVscyA9IHNjYWxlczo6Y29tbWEpICsNCiAgc2NhbGVfY29sb3JfYnJld2VyKHBhbGV0dGUgPSAnUGFpcmVkJykgKw0KICBsYWJzKHggPSAnRGF0ZScsDQogICAgICAgeSA9ICdOZXcgRGFpbHkgQ291bnRzJywgDQogICAgICAgdGl0bGUgPSAnUGxvdHMgb2YgQ29uZmlybWVkIENhc2VzIGFuZCBEZWF0aHMgZm9yIFN0LiBMb3VpcyBNU0EnKSAtPiBwMg0KDQpnZ3Bsb3RseShwMikNCg0KICANCmBgYA0KDQpBbiBpbXBvcnRhbnQgcXVlc3Rpb24gdG8gYWRkcmVzcyAoaWYgdGhpcyBpcyBpbiBQZXRlcidzIGRhdGEpIGlzIGhvdyBkbyB3ZSAqKmhhbmRsZSBuZWdhdGl2ZSBjb3VudHMqKi4gQSBuYWl2ZSBmaXggaW4gUiB3b3VsZCBiZSB0byB1c2UgdGhlIG1heCBvZiB0aGUgdmFsdWUgYW5kIDAgKHNlZSB0aGUgYHBtYXgoKWApLiBIb3dldmVyLCB0aGlzIGlnbm9yZXMgdGhlIG1pc3JlcG9ydGluZyBvZiB0aGUgZGF0YS4gDQoNCi0tLQ0KDQojIENyZWF0aW5nIHRpbWUgc3BsaXRzIGZvciBUcmFpbmluZyBhbmQgVmFsaWRhdGlvbg0KDQpCZWxvdywgSSBoYXZlIGNhcGl0YWxpemVkIG9uIHRoZSBgaW5pdGlhbF90aW1lX3NwbGl0KClgIHRvIHNwbGl0IHRoZSB0aW1lLXNlcmllcyBzdWNoIHRoYXQgdGhlIGZpcnN0IDkwJSBvZiB0aGUgZGF0YSBhcmUgdXNlZCBmb3IgbW9kZWwgYnVpbGRpbmcgYW5kIHRoZSByZW1haW5pbmcgMTAlIGFyZSB1c2VkIGZvciB2YWxpZGF0aW9uLiAqKldlIGNhbiByZWRvIHRoaXMgaWYgd2Ugc29sZWx5IHdhbnQgdG8gZm9jdXMgb24gdGhlIGV4cGxhbmF0b3J5IG1vZGVsaW5nIGNvbXBvbmVudC4qKg0KDQpgYGB7ciB0c19zcGxpdH0NCnNwbGl0cyA9IGluaXRpYWxfdGltZV9zcGxpdChzdF9sb3Vpc19hZ2dfdGJsLCBwcm9wID0gMC45KQ0KDQojIFByaW50aW5nIG91dCB0aGUgc3BsaXRzIHNvIHRoYXQgd2Uga25vdyB0aGUgbnVtYmVyIG9mIG9ic2VydmF0aW9ucyB1c2VkIGZvciBtb2RlbCB0cmFpbmluZw0Kc3BsaXRzDQoNCiMgVHJhaW5pbmcgZGF0YSBzdGFydCBhbmQgZW5kIGRhdGVzDQpwYXN0ZSgnVGhlIHN0YXJ0aW5nIGFuZCBlbmRpbmcgZGF0ZXMgZm9yIHRyYWluaW5nIGFyZScsDQogICAgICBzcGxpdHMkZGF0YVtzcGxpdHMkaW5faWQsICdkYXRlJ10gJT4lIGhlYWQobj0xKSAlPiUgcHVsbCgpLA0KICAgICAgJ2FuZCcsIA0KICAgICAgc3BsaXRzJGRhdGFbc3BsaXRzJGluX2lkLCAnZGF0ZSddICU+JSB0YWlsKG49MSkgJT4lIHB1bGwoKSwNCiAgICAgICdyZXNwZWN0aXZlbHkuIEZvciB0aGUgaG9sZG91dCBkYXRhLCB0aGUgc3RhcnRpbmcgYW5kIHRyYWluaWcgZGF0ZXMgYXJlJywNCiAgICAgIHNwbGl0cyRkYXRhW3NwbGl0cyRvdXRfaWQsICdkYXRlJ10gJT4lIGhlYWQobj0xKSAlPiUgcHVsbCgpLA0KICAgICAgJ2FuZCcsIA0KICAgICAgc3BsaXRzJGRhdGFbc3BsaXRzJG91dF9pZCwgJ2RhdGUnXSAlPiUgdGFpbChuPTEpICU+JSBwdWxsKCkpDQpgYGANCg0KLS0tDQoNCiMgVHJhaW5pbmcgRGlmZmVyZW50IFRpbWUtU2VyaWVzIE1vZGVscw0KDQpJbiB0aGlzIHNlY3Rpb24sIEkgaGF2ZSBxdWlja2x5IHRyYWluZWQgdGhlIGZvbGxvd2luZyBmaXZlIG1vZGVsczogIA0KDQotIEEgdW5pdmFyaWF0ZSBBdXRvIEFSSU1BIG1vZGVsIHdpdGggbm8geHJlZyAgDQotIEFuIEF1dG8gQVJJTUEgbW9kZWwgd2l0aCBjb25maXJtZWQsIGhvbGlkYXlzIChOWVNFIGhvbGlkYXlzKSBhbmQgZG9taW5hbnQgdmFyaWFudCBhcyBvdXIgeHJlZyAgDQotIEFuIEF1dG8gQXJpbWEgTW9kZWwgd2l0aCB4Z2Jvb3N0ICh1c2luZyB0aGUgeHJlZ3MpIG9uIHRoZSBBcmltYSBlcnJvcnMgIA0KLSBUaGUgUHJvcGhldCBNb2RlbCwgb3JpZ2luYWxseSBkZXZlbG9wZWQgYnkgW0ZhY2Vib29rXShodHRwczovL2ZhY2Vib29rLmdpdGh1Yi5pby9wcm9waGV0LykuIFNlZSB0aGUgW0ZvcmVjYXN0aW5nIGF0IFNjYWxlIFBhcGVyXShodHRwczovL3BlZXJqLmNvbS9wcmVwcmludHMvMzE5MC8pIGZvciBtb3JlIGRldGFpbHMuICANCi0gQSBsaW5lYXIgcmVncmVzc2lvbiBtb2RlbCB3aXRoIHByZWRpY3RvcnMgYGFzLm51bWVyaWMoZGF0ZSkgKyBjb25maXJtZWQgKyBob2xpZGF5cyArIGRvbWluYW50X3ZhcmlhbnRgDQoNCkkgaGF2ZSBjYXBpdGFsaXplZCBvbiB0aGUgW21vZGVsdGltZSBwYWNrYWdlIHZpZ25ldHRlXShodHRwczovL2J1c2luZXNzLXNjaWVuY2UuZ2l0aHViLmlvL21vZGVsdGltZS9hcnRpY2xlcy9nZXR0aW5nLXN0YXJ0ZWQtd2l0aC1tb2RlbHRpbWUuaHRtbCkgdG8gcXVpY2tseSBydW4gdGhlc2UgbW9kZWxzLiBOb3RlIHRoYXQgdGhlIGBtb2RlbHRpbWUgQVBJYCBhbHNvIGFsbG93cyBmb3IgcnVubmluZyBvdGhlciBtYWNoaW5lIGxlYXJuaW5nIG1vZGVscyBmb3IgdGltZS1zZXJpZXMgYXBwbGljYXRpb25zLg0KDQojIyBNb2RlbCBCdWlsZGluZw0KDQpgYGB7ciBtb2RlbF9idWlsZGluZywgbWVzc2FnZT1UUlVFfQ0KIyBhIGJhc2ljIHVuaXZhcmlhdGUgQVJJTUEgbW9kZWwgdXNpbmcg4oCcQXV0byBBcmltYeKAnSB1c2luZyBhcmltYV9yZWcoKQ0KIyB1c2luZyB0aGUgbW9kZWx0aW1lIHBrZyB0aGlzIHdpbGwgYXV0b21hdGljYWxseSBwaWNrIHRoZSB3ZWVrbHkgc2Vhc29uYWxpdHkNCm1vZGVsX2ZpdF9hcmltYSA9YXJpbWFfcmVnKCkgJT4lDQogICAgc2V0X2VuZ2luZShlbmdpbmUgPSAiYXV0b19hcmltYSIpICU+JQ0KICAgIGZpdChkZWF0aHMgfiBkYXRlLCBkYXRhID0gdHJhaW5pbmcoc3BsaXRzKSApDQoNCiMgQVJJTUEgd2l0aCB4cmVnDQptb2RlbF9maXRfYXJpbWFfeHJlZyA9IGFyaW1hX3JlZygpICU+JQ0KICBzZXRfZW5naW5lKGVuZ2luZSA9ICJhdXRvX2FyaW1hIikgJT4lDQogICMgY29uZmlybWVkLCBob2xpZGF5cyBhbmQgZG9taW5hbnQgdmFyaWFudCBhcyBvdXIgeHJlZw0KICBmaXQoDQogICAgZGVhdGhzIH4gZGF0ZSArIGNvbmZpcm1lZCArIGhvbGlkYXlzICsgZG9taW5hbnRfdmFyaWFudCwNCiAgICAgIGRhdGEgPSB0cmFpbmluZyhzcGxpdHMpIA0KICAgICkNCg0KIyBBdXRvIEFyaW1hIE1vZGVsIHdpdGggeGdib29zdCBvbiB0aGUgQXJpbWEgZXJyb3JzDQptb2RlbF9maXRfYXJpbWFfYm9vc3RlZCA9IGFyaW1hX2Jvb3N0KA0KICAgIG1pbl9uID0gMiwNCiAgICBsZWFybl9yYXRlID0gMC4wMTUNCikgJT4lDQogICAgc2V0X2VuZ2luZShlbmdpbmUgPSAiYXV0b19hcmltYV94Z2Jvb3N0IikgJT4lDQogICAgZml0KGRlYXRocyB+IGRhdGUgKyBjb25maXJtZWQgKyBob2xpZGF5cyArIGRvbWluYW50X3ZhcmlhbnQsDQogICAgICAgIGRhdGEgPSB0cmFpbmluZyhzcGxpdHMpICkNCg0KIyB0aGUgcHJvcGhldCBtb2RlbCB1c2VkIGJ5IEZhY2Vib29rDQptb2RlbF9maXRfcHJvcGhldCA9IHByb3BoZXRfcmVnKCkgJT4lDQogIHNldF9lbmdpbmUoZW5naW5lID0gInByb3BoZXQiKSAlPiUNCiAgZml0KGRlYXRocyB+IGRhdGUgKyBjb25maXJtZWQgKyBob2xpZGF5cyArIGRvbWluYW50X3ZhcmlhbnQsDQogICAgICBkYXRhID0gdHJhaW5pbmcoc3BsaXRzKSApDQoNCiMgTGluZWFyIFJlZ3Jlc3Npb24gDQptb2RlbF9maXRfbG0gPSBsaW5lYXJfcmVnKCkgJT4lDQogICAgc2V0X2VuZ2luZSgibG0iKSAlPiUNCiAgICBmaXQoZGVhdGhzIH4gYXMubnVtZXJpYyhkYXRlKSArIGNvbmZpcm1lZCArIGhvbGlkYXlzICsgZG9taW5hbnRfdmFyaWFudCwNCiAgICAgICAgZGF0YSA9IHRyYWluaW5nKHNwbGl0cykgKQ0KDQptb2RlbHNfdGJsID0gbW9kZWx0aW1lX3RhYmxlKA0KICAgIG1vZGVsX2ZpdF9hcmltYSwNCiAgICBtb2RlbF9maXRfYXJpbWFfeHJlZywNCiAgICBtb2RlbF9maXRfYXJpbWFfYm9vc3RlZCwNCiAgICBtb2RlbF9maXRfcHJvcGhldCwNCiAgICBtb2RlbF9maXRfbG0NCikNCg0KbW9kZWxzX3RibA0KYGBgDQoNCioqSWYgdGhpcyBkYXRhIGlzIGNsb3NlIHRvIHlvdXIgZGF0YSwgeW91IGNhbiBzZWUgdGhhdCB0aGUgZmlyc3Qgbm9uLXNlYXNvbmFsIGRpZmZlcmVuY2Ugd2FzIHRha2VuIGluIGJvdGggYXV0by5hcmltYSgpIGFwcGxpY2F0aW9ucy4qKiBUaGlzIGFuc3dlcnMgdGhlIHF1ZXN0aW9uIGZyb20gQWxsaXNvbiB0b2RheS4NCg0KDQojIyBUcmFpbmluZyBQZXJmb3JtYW5jZQ0KDQojIyMgUGxvdHRpbmcgdGhlIFJlc2lkdWFscw0KDQpUaGUgY29kZSBiZWxvdyBpcyB1c2VkIHRvIHBsb3QgdGhlIHJlc2lkdWFscyBmb3IgZWFjaCBvZiB0aGUgbW9kZWxzLiBUaGUgZmlyc3QgdGhyZWUgbGluZXMgb2YgY29kZSBhcmUgdmVyeSBzaW1pbGFyIHRvIHRoZSBjb2RlIHNob3duIGluIHRoZSBwYWNrYWdlIHZpZ25ldHRlLiBUaGUgYWRkaXRpb25hbCBjb2RlIGlzIHRvIGFsbG93IHlvdSB0byBjdXN0b21pemUgdGhlIHBsb3QgdG8geW91ciBsaWtpbmcuIA0KDQpgYGB7ciByZXNfdHJhaW5pbmdfdml6fQ0KbW9kZWxzX3RibCAlPiUNCiAgICBtb2RlbHRpbWVfY2FsaWJyYXRlKG5ld19kYXRhID0gdHJhaW5pbmcoc3BsaXRzKSkgJT4lDQogICAgbW9kZWx0aW1lX3Jlc2lkdWFscygpICU+JQ0KICAgIHBsb3RfbW9kZWx0aW1lX3Jlc2lkdWFscyguaW50ZXJhY3RpdmUgPSBGQUxTRSwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLnR5cGUgPSAndGltZXBsb3QnKSArDQogIHNjYWxlX3hfZGF0ZShicmVha3MgPSBzY2FsZXM6OnByZXR0eV9icmVha3MoMTIpKSArDQogIHNjYWxlX2NvbG9yX2JyZXdlcihwYWxldHRlID0gJ0RhcmsyJykgKw0KICBmYWNldF93cmFwKH4gLm1vZGVsX2Rlc2MsIG5jb2wgPSAxLCBzY2FsZXMgPSAnZnJlZScpICsNCiAgdGhlbWVfYncoKSArDQogIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbiA9ICdub25lJykgKw0KICBsYWJzKA0KICAgIHRpdGxlID0gJ1Jlc2lkdWFscyBwbG90IGZvciB0aGUgZml2ZSBtb2RlbHMgYmFzZWQgb24gb3VyIHRyYWluaW5nIGRhdGEnLA0KICAgIHN1YnRpdGxlID0gJ1RoZSByZXNpZHVhbHMgYXJlIGxhcmdlIG9uIHRoZSBzYW1lIGRheSBpcnJlc3BlY3RpdmUgb2YgbW9kZWwnKQ0KDQpgYGANCg0KIyMjIFN0YXRpc3RpY2FsIFRlc3RzIGZvciB0aGUgUmVzaWR1YWxzDQoNClRoZSBgbW9kZWx0aW1lX3Jlc2lkdWFsc190ZXN0KClgIGNvbXB1dGVzIHRoZSByZXN1bHRzIGZyb20gNCBkaWZmZXJlbnQgc3RhdGlzdGljYWwgdGVzdHM6ICANCg0KLSAqKlNoYXBpcm8tV2lsayBUZXN0KiogdGVzdHMgdGhlIE5vcm1hbGl0eSBvZiB0aGUgcmVzaWR1YWxzLiBUaGUgTnVsbCBIeXBvdGhlc2lzIGlzIHRoYXQgdGhlIHJlc2lkdWFscyBhcmUgbm9ybWFsbHkgZGlzdHJpYnV0ZWQuIEEgbG93ICpwLSp2YWx1ZSBiZWxvdyBhIGdpdmVuIHNpZ25pZmljYW5jZSBsZXZlbCBpbmRpY2F0ZXMgdGhlIHZhbHVlcyBhcmUgKipOT1QgTm9ybWFsbHkgRGlzdHJpYnV0ZWQqKi4gIA0KDQotIEJvdGggdGhlICoqQm94LVBpZXJjZSoqIGFuZCAqKkxqdW5nLUJveCoqIFRlc3RzIGFyZSB1c2VkIHRvIHRlc3QgZm9yIHRoZSBhYnNlbmNlIG9mIGF1dG9jb3JyZWxhdGlvbiBpbiByZXNpZHVhbHMuIEEgbG93IHAtdmFsdWUgYmVsb3cgYSBnaXZlbiBzaWduaWZpY2FuY2UgbGV2ZWwgaW5kaWNhdGVzIHRoZSB2YWx1ZXMgYXJlIGF1dG9jb3JyZWxhdGVkLiAgDQoNCg0KYGBge3IgcmVzX3RyYWluaW5nX3Rlc3R9DQptb2RlbHNfdGJsICU+JQ0KICAgIG1vZGVsdGltZV9jYWxpYnJhdGUobmV3X2RhdGEgPSB0cmFpbmluZyhzcGxpdHMpKSAlPiUNCiAgICBtb2RlbHRpbWVfcmVzaWR1YWxzKCkgJT4lDQogIG1vZGVsdGltZV9yZXNpZHVhbHNfdGVzdCgpDQoNCiMgSWYgeW91IHdhbnQgdG8gZXhwbG9yZSB0aGUgcmVzaWR1YWxzIHNlcGVyYXRlbHksIHRoZXkgYXJlIHN0b3JlZCBhcyBmb2xsb3dzDQp0cmFpbmluZ19yZXNpZHVhbHMgPSBtb2RlbHNfdGJsICU+JQ0KICAgIG1vZGVsdGltZV9jYWxpYnJhdGUobmV3X2RhdGEgPSB0cmFpbmluZyhzcGxpdHMpKSAlPiUNCiAgICBtb2RlbHRpbWVfcmVzaWR1YWxzKCkgDQpgYGANCg0KVGhlIHJlc3VsdHMgYWJvdmUgaW5kaWNhdGUgdGhhdCB0aGUgcmVzaWR1YWxzIGFyZSAqKm5vdCBub3JtYWxseSBkaXN0cmlidXRlZCoqLiBIb3dldmVyLCB0aGV5IGFsc28gaW5kaWNhdGUgdGhhdCBubyBzaWduaWZpY2FudCBhdXRvY29ycmVsYXRpb24gaXMgZXhoaWJpdGVkIGluIHRoZSByZXNpZHVhbHMgYXMgaW5kaWNhdGVkIGJ5IGJvdGggdGhlICoqQm94LVBpZXJjZSoqIGFuZCAqKkxqdW5nLUJveCoqIHRlc3QgcmVzdWx0cy4gDQoNCiMjIyBNb2RlbCdzIFRyYWluaW5nIFBlcmZvcm1hbmNlDQoNCk11bHRpcGxlIGFjY3VyYWN5IG1lYXN1cmVzIGNhbiBiZSBjb21wdXRlZCBhcyBmb2xsb3dzLiANCmBgYHtyIHJlc190cmFpbmluZ19hY2N9DQptb2RlbHNfdGJsICU+JQ0KICAgIG1vZGVsdGltZV9jYWxpYnJhdGUobmV3X2RhdGEgPSB0cmFpbmluZyhzcGxpdHMpKSAlPiUNCiAgICBtb2RlbHRpbWVfYWNjdXJhY3koKSAlPiUNCiAgICB0YWJsZV9tb2RlbHRpbWVfYWNjdXJhY3koDQogICAgKQ0KYGBgDQoNCk5vdGUgdGhhdCBNQVBFIGlzIGluZmluaXRlIGluIHRoZSBiZWxvdyB0YWJsZSBzaW5jZSB0aGVyZSBhcmUgc2V2ZXJhbCBkYXlzIHdoZXJlIHRoZSBjb3VudCBvZiBgbmV3IGRlYXRocyA9IDBgLiBUaGUgTUFTRSBpbmRpY2F0ZXMgdGhhdCBhbGwgbW9kZWxzIGltcHJvdmUgb3ZlciB0aGUgbmFpdmUgZm9yZWNhc3QgYnkgYWJvdXQgMjUlLiBIb3dldmVyLCB0aGUgcnNxIGZvciBhbGwgbW9kZWxzIGlzIGxvdy4gKipNYXkgYmUgd2UgbmVlZCB0byBhZGQgYWRkaXRpb25hbCBwcmVkaWN0b3JzKiouIA0KDQoNCi0tLQ0KDQojIFRlc3RpbmcgUGVyZm9ybWFuY2UNCg0KVGhlIG5hbWUgb2YgdGhlIGZ1bmN0aW9uIGZyb20gdGhlIGBtb2RlbHRpbWVgIHBhY2thZ2UgaXMgc29tZXdoYXQgY29uZnVzaW5nLiBJdCBpcyBjYWxsZWQgYGNhbGlicmF0ZWAgYnV0IGNhbGlicmF0aW5nIGFkZHMgYSBuZXcgY29sdW1uLCBgLmNhbGlicmF0aW9uX2RhdGFgLCB3aXRoIHRoZSB0ZXN0IHByZWRpY3Rpb25zIGFuZCByZXNpZHVhbHMgaW5zaWRlLiBBIGZldyBub3RlcyBvbiBDYWxpYnJhdGlvbjogIA0KDQotIENhbGlicmF0aW9uIGlzIGhvdyBjb25maWRlbmNlIGludGVydmFscyBhbmQgYWNjdXJhY3kgbWV0cmljcyBhcmUgZGV0ZXJtaW5lZCAgDQotIENhbGlicmF0aW9uIERhdGEgaXMgc2ltcGx5IGZvcmVjYXN0aW5nIHByZWRpY3Rpb25zIGFuZCByZXNpZHVhbHMgdGhhdCBhcmUgY2FsY3VsYXRlZCBmcm9tIG91dC1vZi1zYW1wbGUgZGF0YS4gICANCi0gQWZ0ZXIgY2FsaWJyYXRpbmcsIHRoZSBjYWxpYnJhdGlvbiBkYXRhIGZvbGxvd3MgdGhlIGRhdGEgdGhyb3VnaCB0aGUgZm9yZWNhc3Rpbmcgd29ya2Zsb3cuDQoNCg0KYGBge3IgcmVjbGFpYnJhdGlvbn0NCmNhbGlicmF0aW9uX3RibCA9IG1vZGVsc190YmwgJT4lDQogICAgbW9kZWx0aW1lX2NhbGlicmF0ZSggbmV3X2RhdGEgPSB0ZXN0aW5nKHNwbGl0cykgKQ0KDQpjYWxpYnJhdGlvbl90YmwNCmBgYA0KDQpQZXIgdGhlIGRlc2NyaXB0aW9uIGFib3ZlIHRoZSBjb2RlIGNodW5rLCB0aGUgbW9kZWwgZGVzY3JpcHRpb25zIGRpZCBub3QgY2hhbmdlIGFuZCB3ZSBvbmx5IGFwcGVuZGVkIHRoZSB0ZXN0aW5nIGRhdGEuDQoNCg0KIyMgSG9sZG91dCBQZXJmb3JtYW5jZQ0KDQojIyMgUGxvdHRpbmcgdGhlIFJlc2lkdWFscw0KDQpgYGB7ciByZXNfaG9sZG91dF92aXp9DQptb2RlbHNfdGJsICU+JQ0KICAgIG1vZGVsdGltZV9jYWxpYnJhdGUobmV3X2RhdGEgPSB0ZXN0aW5nKHNwbGl0cykpICU+JQ0KICAgIG1vZGVsdGltZV9yZXNpZHVhbHMoKSAlPiUNCiAgICBwbG90X21vZGVsdGltZV9yZXNpZHVhbHMoLmludGVyYWN0aXZlID0gRkFMU0UsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgIC50eXBlID0gJ3RpbWVwbG90JykgKw0KICBzY2FsZV94X2RhdGUoYnJlYWtzID0gc2NhbGVzOjpwcmV0dHlfYnJlYWtzKDEyKSkgKw0KICBzY2FsZV9jb2xvcl9icmV3ZXIocGFsZXR0ZSA9ICdEYXJrMicpICsNCiAgZmFjZXRfd3JhcCh+IC5tb2RlbF9kZXNjLCBuY29sID0gMSkgKw0KICB0aGVtZV9idygpICsNCiAgdGhlbWUobGVnZW5kLnBvc2l0aW9uID0gJ25vbmUnKSArDQogIGxhYnMoDQogICAgdGl0bGUgPSAnUmVzaWR1YWxzIHBsb3QgZm9yIHRoZSBmaXZlIG1vZGVscyBiYXNlZCBvbiBvdXIgaG9sZG91dCBkYXRhJywNCiAgICBzdWJ0aXRsZSA9ICdUaGUgQVJJTUEgdHlwZSBtb2RlbHMgYXJlIG1vc3QgYWNjdXJhdGUuIExNIGlzIG92ZXItcHJlZGljdGluZyBzaW5jZSByZXNpZHVhbHMgYXJlIG1vc3RseSBuZWdhdGl2ZScpIA0KDQpgYGANCg0KIyMjIFN0YXRpc3RpY2FsIFRlc3RzIGZvciB0aGUgUmVzaWR1YWxzDQoNCmBgYHtyIHJlc19ob2xkb3V0X3Rlc3R9DQptb2RlbHNfdGJsICU+JQ0KICAgIG1vZGVsdGltZV9jYWxpYnJhdGUobmV3X2RhdGEgPSB0ZXN0aW5nKHNwbGl0cykpICU+JQ0KICAgIG1vZGVsdGltZV9yZXNpZHVhbHMoKSAlPiUNCiAgbW9kZWx0aW1lX3Jlc2lkdWFsc190ZXN0KCkNCg0KYGBgDQoNClRoZSBob2xkb3V0IHJlc3VsdHMgYXJlIGNvbnNpc3RlbnQgd2l0aCB0aGUgb2JzZXJ2YXRpb25zIG1hZGUgb24gdGhlIHRyYWluaW5nIGRhdGEuIE9uZSBleGNlcHRpb24gaXMgdGhhdCB0aGUgQVJJTUEgd2l0aCB4cmVnIChtb2RlbCAyKSBzaG93cyBjb3JyZWxhdGVkIHJlc2lkdWFscyBvbiB0aGUgaG9sZG91dCBkYXRhc2V0Lg0KDQoNCiMjIyBNb2RlbCdzIEhvbGRvdXQgUGVyZm9ybWFuY2UNCg0KTm90ZSB0aGF0IE1BUEUgaXMgaW5maW5pdGUgaW4gdGhlIGJlbG93IHRhYmxlIHNpbmNlIHRoZXJlIGFyZSBzZXZlcmFsIGRheXMgd2hlcmUgdGhlIGNvdW50IG9mIGBuZXcgZGVhdGhzID0gMGANCmBgYHtyIHJlc19ob2xkb3V0X2FjY30NCm1vZGVsc190YmwgJT4lDQogICAgbW9kZWx0aW1lX2NhbGlicmF0ZShuZXdfZGF0YSA9IHRlc3Rpbmcoc3BsaXRzKSkgJT4lDQogICAgbW9kZWx0aW1lX2FjY3VyYWN5KCkgJT4lDQogICAgdGFibGVfbW9kZWx0aW1lX2FjY3VyYWN5KA0KICAgICkNCmBgYA0KDQpUaGUgZmlyc3QgYW5kIHRoaXJkIG1vZGVscyBjb250aW51ZSB0byBiZSB0aGUgYmVzdCBtb2RlbHMuIEhvd2V2ZXIsIHRoZWlyIGltcHJvdmVtZW50IG92ZXIgdGhlIG5haXZlIGZvcmVjYXN0IChpLmUuLCByYW5kb20gd2FsaykgaXMgc21hbGxlciB0aGFuIHRoZSB0cmFpbmluZyBkYXRhc2V0Lg0KDQoNCg==